The FM-Index: A Compressed Full-Text Index Based on the BWT

In this talk we address the issue of indexing compressed data both from the theoretical and the practical point of view. We start by introducing the FM-index data structure [2] that supports substring searches and occupies a space which is a function of the entropy of the indexed data. The key feature of the FM-index is that it encapsulates the indexed data (self-index) and achieves the space reduction at no significant slowdown in the query performance. Precisely, given a text T [1, n] to be indexed, the FM-index occupies at most 5nHk(T )+o(n) bits of storage, where Hk(T ) is the k-th order entropy of T , and allows the search for the occ occurrences of a pattern P [1, p] within T in O(p + occ log n) time, where > 0 is an arbitrary constant fixed in advance. The design of the FM-index is based upon the relationship between the Burrows-Wheeler compression algorithm [1] and the suffix array data structure [9]. It is therefore a sort of compressed suffix array that takes advantage of the compressibility of the indexed data in order to achieve space occupancy close to the Information Theoretic minimum. Indeed, the design of the FM-index does not depend on the parameter k and its space bound holds simultaneously over all k ≥ 0. These remarkable theoretical properties have been validated by experimental results [3, 4] and applications [7, 10]. In particular it has been shown that the FM-index achieves a space occupancy close to the best known compressors and, unlike them, it allows to search for arbitrary substrings in a hundred of megabytes within few millisecs, since it does not decompress the whole file. We will conclude the talk by sketching two intriguing variants of the FM-index. One achieves O(p + occ) query time (i.e. output sensitivity) and uses O(nHk(T ) log n) + o(n) bits of storage. This data structure exploits the interplay between two compressors: the Burrows-Wheeler algorithm and the LZ78 algorithm [11]. Our other proposal [8] combines two recent and elegant techniques—the compression boosting [5] and the wavelet tree [6]—to design a variant of the FM-index that scales well with the size of the input alphabet.